As shown in the left experiments, for a given target pose, adjusting the composition of the same human in the reference image ( i .

As shown in the left experiments, for a given target pose, adjusting the composition of the same human in the reference image ( i .

3D human shape reconstruction under severe occlusion due to human-object or human-human interaction is a challenging problem. Parametric models i.e., SMPL(-X), which are based on the statistics across human shapes, can represent whole human body shapes but are limited to minimally-clothed human shapes. Implicit-function-based methods extract features from the parametric models to employ prior knowledge of human bodies and can capture geometric details such as clothing and hair. However, they often struggle to handle misaligned parametric models and inpaint occluded regions given a single RGB image. In this work, we propose a novel pipeline, MHCDIFF, Multi-hypotheses Conditioned Point Cloud Diffusion, composed of point cloud diffusion conditioned on probabilistic distributions for pixel-aligned detailed 3D human reconstruction under occlusion. Compared to previous implicit-function-based methods, the point cloud diffusion model can capture the global consistent features to generate the occluded regions, and the denoising process corrects the misaligned SMPL meshes. The core of MHCDIFF is extracting local features from multiple hypothesized SMPL(-X) meshes and aggregating the set of features to condition the diffusion model. In the experiments on CAPE and MultiHuman datasets, the proposed method outperforms various SOTA methods based on SMPL, implicit functions, point cloud diffusion, and their combined, under synthetic and real occlusions. Our code is publicly available at https://donghwankim0101.github.io/projects/mhcdiff/ .

Modeling 3D human shape is important for many human-centric tasks, such as pose estimation and body shape fitting. However, further research is needed for applications involving dynamic signals, e. g. 3D moving humans. It combines a linear prior model with residual encoded in a learned auxiliary code. Each temporal sequence of 3D human shapes is encoded with compact latent codes, which then can be used to reconstruct the input sequence through a decoder. The additional auxiliary latent code compensates for the inaccurate motion and enriches the geometry details.